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ABSTRACT 

Project-based learning using online learning environments is becoming increasingly popular. To in-depth explore 
the behavioural patterns and limitations faced by students in project-based learning where online forums are 
used. This study conducted an empirical case study of an online project-based learning activity in a management 
course, in which 70 college students participated. The study integrates two analytical approaches: quantitative 
content analysis and progressive sequential analysis. The analytical approaches allow researchers to explore and 
visualize the content and patterns of participants’ online project learning discussion behaviours. The results 
indicate that learners’ limitations include poor time-management and insufficient initial data collection. The 
students also demonstrated a lack of proper data evaluation processes and sufficient comprehensive analysis. The 
limitations are in-depth discussed and suggestions for teachers and educational software developers are also 
proposed. 
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INTRODUCTION 

Project-based learning is a learner-centred instructional approach and can help students develop skills such as 
data collection, coordination, peer discussion, and information analysis (Blumenfeld et al., 1991; Thomas et al., 
1999; Thomas, 2000), and this teaching approach is becoming increasingly applied. Project-based learning has 
been combined with different instructional strategies in recent years and applied to various subject domains (e.g., 
$ahin, 2008; Neo & Neo, 2010; van Rooij, 2009; Hou et al., 2007a). Its characteristics have also been discussed 
in numerous studies (e.g., Koo et al., 2009; Ipek, 2010; Park & Ertmer, 2008; Kramer et al., 2007; Hou et al., 
2007a; Lam et al., 2009; Thomas, 2000; Edelson et al., 1999; Krajcik, et al, 1998). 

The employment of web environments in project-based learning has also become an important aspect of 
educational technology. Past studies have revealed students often lack in-depth analysis process (Hou et al., 
2007a; Hou et al., 2007b; Thomas, 2000; Krajcik, et al., 1998; Edelson, Gordon, & Pea, 1999) and treat online 
information as direct answers to questions on project topics (Wallace & Kupperman, 1997; Chang & McDaniel, 
1995) in project-based learning, and this may result in inappropriate inferences and discussions (Krajcik, et al., 
1998). Limitations in peer collaboration and time management have also been noted (Thomas, 2000; Hou et al., 
2007b). 

On the other hand, since asynchronous discussion allows learners to interact with each other on a topic and 
construct knowledge, argumentation may also be achieved (e.g. Oh & Jonassen, 2007; Cho & Jonassen, 2006). 
Due to the increasing popularity of online forums, project-based learning that is based on online discussion 
activity (e.g., Hou et al., 2007a) has become a topic that deserves to be analyzed at a deeper level. 

Therefore, the analysis of actual behaviours in project-based learning that incorporates online discussion is the 
key objective in this study. Analyzing the behavioural patterns may further explore the potential causes and 
natures of the aforementioned limitations. These findings may also serve as important references for teachers 
with regard to strategies used for intervention and guidance in project-based learning that incorporates online 
forums. 

There have been studies in the area of online discussion-based teaching (e.g., Hewitt, 2005; Patricia & Dabbagh, 
2005; Hou et al., 2007a, 2008). However, studies on project-learning discussion behaviours (e.g., the 
clarification of topics, coordination of projects, and collection/evaluation of data) are rare recently. Therefore, it 
is a crucial research topic to empirically analyze the behavioural patterns of a project-based learning activity 
incorporating an online discussion activity. In this study, an new analytical approach which integrates two 
analytical methods: quantitative content analysis (Rourke & Anderson, 2004) and sequential analysis (Bakeman 
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& Quera, 1995) is proposed and applied to explore the behavioural patterns of a online project-based learning 
activity. A coding scheme for learning behaviours has been devised based on past studies of project-based 
learning (Blumenfeld et al., 1991; Thomas, Mergendoller & Michaelson, 1999; Thomas, 2000). Each discussion- 
message will be chronologically ordered and will undergo a quantitative content analysis and a series of 
progressive sequential analysis in an integrative manner. The proposed new analytical methods will allow 
researchers to use progressive, chronologically-ordered sequential analysis to infer visualized patterns in the 
progress of students’ project-learning related behaviours. By the above pattern analysis, researchers can in-depth 
explore and discuss limitations of learners’ behavioural patterns. The specific purposes of this research are as 
follows; 

1. Conduct a project-based learning activity incorporating an online discussion activity. Integrate 
quantitative content analysis and progressive sequential analysis to analyze the behavioural patterns of 
learners’ project-learning behaviours. 

2. Discuss limitation of the patterns and provide suggestions for teachers with regard to strategies used for 
guidance in online project-based learning. 

METHOD 

Participants and Procedures 

The participants in this case study were 70 college students studying management, all of whom had completed 
information-related courses and were capable of using online forums. This study arranged a project-based 
learning activity incorporating online discussion in a management course that was undertaken by these students. 
The students were asked to analyze a business-management case assigned by the teacher, and to complete a 
project report collaboratively. Their assignment was to analyze the obstacles faced by an organization’s managers 
and to provide suggestions. The entire project-based discussion activity was conducted in the forum that this 
study provided, and which only had basic functions, such as posting and replying to articles and showing the list 
of articles, in order to avoid interference from outsiders or the complications of a complex interface. The 14-day 
activity required the 70 students to collaborate, jointly collect data and make comments on its analysis, compile 
their findings, and present the final report in the forum by posting articles and responses. During the entire 
process of discussion, the teacher was not allowed to intervene or provide guidance in order to avoid the 
teacher’s subjective guidance. 

Design 

In this study, quantitative content analysis and progressive sequential analyses were adopted. According to the 
coding scheme for project-based learning discussion behaviours (see the section of Coding scheme and Data 
Analysis), each article and response posted by a student was coded based on chronological order so that 
researchers might explore the patterns of students’ daily project-based learning. Lag sequential analysis 
(Bakeman & Quera, 1995) allowed researchers to examine whether the sequence of a certain coded behaviour 
followed by another occurs with statistical significance (e.g., whether the students demonstrated the behavioural 
pattern of conducting an “initial data-evaluation” after completing “data-collection” in a project-based learning 
activity). This method, which has been adopted by studies of online learning and online communities (e.g., Hou 
et al., 2007a, 2008, 2009), allowed researchers to infer a visualized diagram of behavioural patterns. In this 
study, a progressive sequential analysis was proposed and adopted, in which the data that was coded daily 
underwent a lag sequential analysis for behavioural patterns. For example, the codes accumulated in the first day, 
in the first two days, in the first three days, and so on, right up till the last day, were analysed for behavioural 
patterns as a way for exploring how the students’ behaviours shifted on a daily basis. This approach allows us to 
understand the time points at which significant sequences emerge, continue, or disappear. This study proposed 
this fairly new analytical approach with the desire to analyse characteristics and limitations in students’ project- 
based learning, and to increase its research validity. In contrast with past approaches of only conducting one-time 
sequential analyses of all the coded data (e.g., Hou et al., 2007a, 2008, Jeong, 2003), our approach could allow 
us to better understand students’ processes in daily project learning. 

Coding scheme and Data Analysis 

After reviewing past studies on project-based learning (e.g., Blumenfeld et al., 1991; Thomas, Mergendoller & 
Michaelson, 1999; Thomas, 2000, etc.), the study compiled procedures and models for project-based learning. 
This research also considered the characteristics of online discussion and proposed the coding scheme (seen in 
Table 1), which is dedicated to behaviours in online project-based learning discussions. 
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Tab le 1 Coding scheme for project-based learning online discussion behavi ours 

Code Dimension 

PLl Initial analysis of the project topic 

PL2 Presence of data collection relevant to the project. 

PL3 Initial evaluation of the collected data. 

PL4 Writing on and analysis of the project content. 

PL5 Comprehensive analysis, compilation of existing 

analysis and data. 

PL6 Proposing comments regarding task coordination. 

PL7 Discussions irrelevant to the project. 


For content analysis, each article or response posted in the forum was coded hy a professional psychology 
researcher. Since each posted article could receive multiple responses, this study treated each article as a unit: the 
content of a posted article was treated as the first message, while the responses it received were treated as its 
continuing messages based on their chronological order. Using the coding scheme, the coder coded each message 
based on the dominant content of the message and on which item in the coding scheme best applied to the 
content of the message. The codes were chronologically ordered so that researchers could analyze daily 
frequencies. After the discussion from 14 days of activity was coded, 180 coded messages were yielded. To 
ensure the inter-coder reliability, the entire discussion was coded by another coder with a similar background. 
The Kappa value was 0.73 (p < .001), reaching the .001 level of significance. The collected data was then 
provided for further analysis. 

RESULTS AND DISCUSSION 

The distribution of the coded behaviours gathered from the 180 coded messages generated during the 14-day 
activity is shown in Figure 1. Instances of PL3 and PL6 were not found and are thus not included in the figure. 
Figure 1 indicates that PL4 (writing and analysis) is most frequent (67.78%), followed by PLl (initial analysis: 
18.33%). The frequencies of the other codes are much lower than that of PL4. The fact that PL7 (irrelevant 
discussions) only took up 2.22% of the total suggests that the students demonstrated a certain level of focus 
during the project-based learning activity. 



Figure I Distribution of the quantitative eontent anaiysis of codes in oniine project-based learning 

Past research has highlighted learners’ limitations of analysis process in online project-based learning (Hou et 
ah, 2007a; Thomas, 2000; Krajcik, et ah, 1998; Edelson, Gordon, & Pea, 1999). The coding of the behaviours in 
our study may allow in-depth understanding how these limitations are formed. For example, a fact worth 
noticing is that the frequency of PL2 is rather low (8.89%), and the participants may have proceeded directly to 
initial analysis and writing (PL4) without going through data-e valuation (PL3 was not found). In addition, they 
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did not discuss task coordination (PL6 was not found). Compilation and comprehensive analyses were also 
limited (PL5 only took up 2.78%). PL3 and PL5, which indicate the ability to evaluate and summarize different 
comments and information, are important components of argumentation (Erduran et al, 2004) which the 
participants often lacked. 

In order to further examine the issue of time management in project-hased learning as mentioned in previous 
studies (e.g., Thomas, 2000; Hou et ah, 2007h), the number of daily discussions in our study is summarized in 
Figure 2. As shown by Figure 2, the total number of postings from Day 1-8 (taking up 57.14% of the entire task- 
time) was as low as 27.8%. On Day 9, however, the number increased significantly (17.8%), but dropped again 
by Day 10. Aecording to the review of the discussion content on Day 8-10, this temporary rise may have been a 
chain reaction by numerous participants who actively posted articles on Day 9, but the effect was not sustained. 



Figure 2 Distribution of the frequency of daiiy discussions 

After Day 11, however, the extent of discussion nearly doubled every day, indicating a sudden increase as the 
project came to an end. A possible reason for this is that students often procrastinate about their studies until the 
last minute. To better reveal potential causes and more details, this study conducted a quantitative eontent 
analysis based on the time-line. The daily frequency of each code is summarized in Figure 3. The results showed 
that diseussion in the first 8 days was not only less frequent but also had fewer instances of PLl (initial analysis) 
or PL2 (data-collection). The behaviours in the first 8 days were mostly direct analyses and writing on the 
project (PL4). By going directly to PL4, the students may have indicated a lack of initial understanding and data- 
collection, as a result of whieh the depth of the diseussion later on may have been rather shallow, or may have 
involved inappropriate inferences (e.g., Krajcik, et ah, 1998). This study also discovered that PL4 had three ups 
and downs from Day 1 to Day 8, yet its frequeney tends to drop. This indicate that aetive participation in 
analyses during this stage tends to drop and the participants were not motivated enough. On Day 9, PLl and PL2 
increased along with PL4; from Day 12 to Day 14, PLl increased, while data-eollection (PL2) did not. 
Researchers also noted that PL5 (comprehensive analysis) was still rather infrequent during the entire process, 
even in the later stages. This reflects the consequenees of the participants’ lack of initial data collection; as a 
result, they were not able to quickly gather, analyze or compile the needed information and comments from each 
other in the later stages. 
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Figure 3 Frequency of daily codes in online project-based learning 

To better understand the relationships between each code, a series of lag sequential analysis (Bakeman & Quera, 
1995) were conducted, which allows us to examine the overall interactions between codes within a certain time- 
line and to visualize behavioural sequences. The Adjusted Residuals Table of sequential analysis could be 
created by calculating the transition-frequencies of the behaviours during a certain time period (Bakeman & 
Quera, 1995; Hou et al., 2007a, 2008; Jeong, 2003). 

Table 2 is an example (Day 1-9). Each z-score in the table indicates whether a given sequential continuity was 
significant. The rows in the table represent the starting behaviours, and the columns represent behaviours that 
followed the starting behaviours. A z-score of +1.96 or higher indicates a significant sequence (p<0.05). Table 2 
indicates that up until Day 9, the two sequences that reached the level of significance were PL1->PL1 and PL4- 
>PL2. 


Table 2 Adjusted Residuals Table of sequential analysis (lag^l, Day 1-9) 


z 

PLl 

PL2 

PL4 

PL5 

PL7 

PLl 

6.51* 

-0.9 

-1.51 

-0.34 

-0.59 

PL2 

-0.68 

-0.9 

0.24 

-0.34 

1.49 

PL4 

-7.08 

3.05* 

1.01. 

1.15 

-0.72 

PL5 

0 

0 

0 

0 

0 

PL7 

-0.5 

-0.67 

0.61 

-0.25 

-0.44 


p< 0.05 


Though past studies have addressed the sequences of behavioural patterns in online learning (e.g., Hou et al., 
2007a, 2008; Jeong, 2003), all of them have been a one-time analyses in which all the codes were gathered and 
analysed together. This approach, however, does not allow us to explore the possible progressive transformations 
of behavioural pattern sequences in each time stage. To address this issue, this study conducted a progressive 
sequential analysis; that is, the codes accumulated from Day 1 through any other day were placed under 
sequential analysis based on their dates (as shown in Table 3). 

The result indicates that the 14-day process can be divided into 3 stages. Based on the significant sequences, the 
behavioural transition diagrams were inferred (as shown in Figure 4); in these, each arrow indicates the direction 
of a shift in a significant sequence. Based on this, researchers can further explore how the discussion behaviours 
are correlated with each other. 


Copyright © The Turkish Online Journal of Educational Technology 


56 



TOJET: The Turkish Online Journal of Educational Technology - July 2010, volume 9 Issue 3 


Table 3 Progressive sequential analysis of diseussion behaviours in online pr ojeet-based learning 

Day Significant Behavioural Sequences 


1 

N/A 

1-2 

N/A 

1-3 

N/A 

1-4 

N/A 

1-5 

N/A 

1-6 

N/A 

1-7 

N/A 

1-8 

N/A 

1-9 

PL1->PL1, PL4->PL2 

1-10 

PL1->PL1,PL4->PL2 

1-11 

PL1->PL1,PL4->PL2 

1-12 

PL1->PL1, PL4->PL2, PL4->PL4 

1-13 

PL1->PL1, PL4->PL2, PL4->PL4 

1-14 

PL1->PL1, PL4->PL2, PL4->PL4 



Day 12-14 

Figure 4 Progressive sequential-analysis transition diagrams in eaeh stage 


This study diseovered that in Stage 1 (Day 1-8) not only is the extent of discussion low, as revealed hy the 
quantitative content analysis, hut there is also no significant sequential correlation between these behaviours. In 
Stage 2 (Day 9-11), the participants started showing continuity in initial analysis (PL1->PL1) and analysis with 
data collection (PL4->PL2). In the last stage. Stage 3 (Day 12-14), the participants further demonstrated 
continuity in analysis and report-writing (PL4->PL4). These sequences indieate the behaviours in Stage 1 are 
more sporadic and separated from one another, and they take up about 60% of the entire discussion time. 
Although initial analysis is sustained in Stage 2 (PL1->PL1), the appearance of the sequence PL4->PL2 indicates 
that most of the participants started to collect data after an official analysis and did not follow the expeeted 
course, such as initial analysis -> data collection -> data evaluation -> report-writing (e.g., neither PL1->PL2, 
PL2->PL3, PL3->PL4, nor PL2->PL4 have reaehed a level of significance). This finding reveals the detail 
information regarding the limitations in learners’ information analysis (e.g., Wallace & Kupperman, 1997; 
Chang & MeDaniel, 1995); that is, they looked for information they needed for writing a report without first 
going through a systematic information analysis process, and they may have directly applied online information 
to offieial analysis and reports without proper evaluation. In Stage 3, although analysis and report-writing show 
continuity, they still had not reached the significance of the sequence “PL4->PL5” (comprehensive analysis) 
before the entire learning aetivity was over. The quantitative content analysis indieates that the insufficient 
information gathered may have limited the depth of discussion. Information on the categorization of these stages, 
sequenees, and frequeney, eould serve as references for teaeher intervention. 

The above analyses demonstrates the limitations of students in online project-based learning in terms of time 
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management and information analysis, and may allow us to in-depth realize some possible causes, inter- 
behaviour interactions, and those areas that require more guidance from teachers. 

SUGGESTIONS 

In this case study, the empirical observation and analyses indicate that participants’ limitations include poor time 
management (procrastinating until the last moment) and insufficient initial data collection. The students also 
demonstrated a lack of proper data evaluation processes (e.g., lack of PL1->PL2, PL2->PL3, PL3->PL4, PL2- 
>PL4, or PL4->PL5, etc) and sufficient comprehensive analysis (PL5). One phenomenon observed in regard to 
time management was decreased motivation to continue a given discussion, which may lead to the effect of 
“dead discussion-threads” as proposed by Hewitt (2005). With little time left in Stage 3, the depth of analysis 
may have also been hindered. Another important issue is that the participants did not conduct initial data- 
evaluation and talk about task-coordination (PL3 and PL6 are absent), and therefore the discussion may lack a 
structure or appropriate order. 

The above findings could help teachers and researchers gain a better understanding of the relevant limitations in 
online project-based learning, and may serve as valuable references for timely interventions. This study then 
provides suggestions for teachers and educational system developers are as follows: 

1 . A teacher may motivate students and increase the extent of discussion by providing incentives in the 
initial stages of the course. To avoid the abovementioned limitation in the sequence PL4->PL2, the teacher 
may post an article to remind learners not to neglect data collection (PL2) or initial data evaluation (PL3). This 
may also lead to better information analysis and compilation (such as triggering the sequences PL1->PL2, 
PL2->PL3, PL3->PL4, PL2->PL4, or PL4->PL5). The teacher could also consider using scaffolding (e.g., van 
Rooij, 2009) by asking students to complete certain analytical procedures in a given order and review their 
coordination and task delegation. As for the chain-reaction caused by numerous participants actively posting 
articles, and its sustainability, the teacher is advised to notice the community’s “peripheral members” (Zhang 
& Storck, 2001) and to post articles that encourage them to participate in the discussion right from the initial 
stages, and to interact with other members. Since the “community climate” may also affect knowledge sharing 
(Bock et ah, 2005), the teacher could post articles to break the ice or settle conflicts in coordination. It is hoped 
that the above recommendations can serve as valuable points of reference for teacher intervention. 

2. To a developer of educational systems, the limitations in students’ co-operative project-based learning 
serve as important references for the development of an automated detection system. The topic of embedding 
sequential analysis detection calculation into an e-leaming system deserves to be explored (Hou et ah, 2010), 
since the result could be an automation for the instant analysis of the frequencies and sequences of students’ 
behaviours. This system may also serve as a reference for teachers’ dynamic guidance and intervention. 

CONCLUSION 

This study aims to in-depth analyze the behavioural patterns of learners’ project-learning behaviours. This study 
applied two analytical approaches: quantitative content analysis and progressive sequential analysis to explore 
the behavioural patterns of learners’ online discussions. The results indicate some learners’ limitations in their 
project-based learning activity. The limitations include poor time management and insufficient initial data 
collection. The students also demonstrated a lack of proper data evaluation processes (e.g., lack of PL1->PL2, 
PL2->PL3, PL3->PL4, PL2->PL4, or PL4->PL5) and sufficient comprehensive analysis (lack of PL5), the 
suggestions for teachers and educational software developers are also proposed in the above suggestion section. 
Regarding the limitations of the study, due to the limitations of the samples, this study focused more on the 
project-based learning in higher education settings. This may limit generalization of findings to other grade years 
or subjects. More empirical studies on different academic systems and subject domains may be explored in the 
future in order to determine the similarities, differences of the behavioral patterns. Moreover, the above proposed 
specific framework and techniques of automatic detection environment for online project-based learning 
discussion are yet to be developed and explored by future studies. 

Finally, integrating more interactive learning strategies (e.g., problem-solving, peer assessment, role playing or 
peer tutoring) with project-based learning instructional activities may provides scaffolding functions to enhance 
learners’ knowledge construction, and researchers may apply more coding schemes (e.g., social knowledge 
construction, problem-solving process or cognitive phases) at the same time to explored the discussion content 
and patterns in project-based learning for higher validity in future studies. 

It is wished that the analytical methods (i.e., quantitative content analysis with progressive sequential analysis), 
findings, and suggestions in this study can serve as references for teachers, researchers, and system developers 
working on online project-based learning. 
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